Skip to content

feat: a Device tab shows a live iOS or Android simulator, and agents can drive it - #1010

Merged
badcuban merged 55 commits into
mainfrom
feat/ios-simulator-setup
Nov 20, 2026
Merged

badcuban merged 55 commits into
mainfrom
feat/ios-simulator-setup

Conversation

@badcuban

@badcuban badcuban commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Users building mobile apps had no way to see or touch an iOS Simulator or Android Emulator from Threadlines, and agents had no supported way to drive one while the user watches.

This adds a Device tab in the right panel, ported from T3 Code's Device panel (MIT, credited in NOTICE) and taken past it in places. Design: docs/design/devices.md.

What users get

  • A live, tappable simulator or emulator from the computer running the Threadlines server: typing, paste, hardware buttons, screenshot, power off, and a Keyboard button on touch screens.
  • Model-accurate device frames drawn in code: screen radius, Dynamic Island, notch or punch hole, side buttons, and the iPhone Duo's hinge. No Apple artwork; numbers come from Xcode's device descriptions.
  • iPhone Duo with both screens and stance controls (closed, book, open, laptop, tent, table). Touches only reach the display in front, including a finger held through a handoff. Android foldables fold and unfold.
  • A Tools drawer: appearance, text size, accessibility toggles, location, permissions, open URL, launch or quit an app, shake, test push, the app in front, and an iOS event log. Every value is read back from the device.
  • An accessibility-frame overlay with an inspect mode, where a tap shows the label instead of touching the device.
  • Several devices open at once as chips, and a device that floats over the chat and can be docked again.
  • Record a flow. The step log (taps named from the accessibility tree), screenshots and a video under 10 MiB are added to the thread's unsent draft. Anything that can't be saved stays downloadable, and nothing already in the draft is replaced.
  • Settings → Devices: "Use devices" and "Agents can use devices" (both off by default) and tool versions.
  • Works in the desktop app, the web app, the phone layout and on saved computers over the relay.

What agents get

  • device_list, device_open, device_screenshot, device_close on a new threadlines_device MCP server, registered for Codex, Claude, ACP, OpenCode, Pi and Muse.
  • device_open returns a launcher bound to the thread's agent-device session, and the Device tab opens for anyone watching.
  • Never for fenced threads, side runtimes or sessions limited by outside work. A physical phone is never picked by default.

How it works

  • The server owns devices. It installs expo-device-hub 0.22.1 and agent-device 0.21.36 at pinned versions into its state dir and runs them on Threadlines' managed Node.
  • The hub binds loopback with --require-token; only the server holds the token.
  • Video and input ride the client's existing WS RPC connection through a device channel with a path allowlist, server-added auth, credit flow control and a 16 MiB per-frame cap. The preview tunnel refuses device-tool ports.
  • Device claims survive restarts and are released on archive, move and delete.
  • The hub start runs once in the service, so closing a tab can't strand an install.
  • The server finds Java for the Android SDK tools (JAVA_HOME, Android Studio's own, or an installed JDK) and says when there is none.
  • DeviceSummary.model carries the simulator's device type, so frames never depend on a renamed simulator's name.
  • Draft saving: an attachment too big for local storage no longer fails the whole draft save. Attachments already saved keep their data, and a failed write never un-saves earlier attachments.

Left out on purpose

  • Expo cloud simulators (billed), Simlock pools, SSH device hosts.
  • T3's 3D models: they are Apple's AR assets with no redistribution license.
  • WebRTC.
  • A keyboard shortcut. Older clients and servers reject the whole keybindings file over an unknown command, so "Open device" is in the command palette instead.

Worth noting

  • When both device switches are on, Claude's standing instructions grow by 711 bytes.
  • ExecutionEnvironmentCapabilities.devices hides everything for older servers.

Checked

  • vp fmt, vp lint and vp run typecheck are clean.
  • Server 4,305, web unit 3,317, web browser 1,147, client-runtime, shared 692 and contracts 117 tests pass.
  • Live on macOS with isolated servers: the iOS 27.2 stream, taps, the Tools drawer, accessibility frames, chips, the float, recording, the event log, first-start listing, a Pixel Fold emulator folding, and power off.
  • The design and the diff were reviewed independently in several rounds; every finding was fixed or is listed below.

Not checked live

  • The iPhone Duo: the only iOS runtime here (27.2, build 24B8117) lists the Duo as unsupported. It is covered by a stand-in helper in a browser test.
  • A real agent session driving a simulator, the relay, and Windows.

Follow-ups (already on main, not from this PR)

  • Two tabs can overwrite each other's drafts.
  • Switching threads mid-paste can lose the paste.
  • Zero-byte files drop out of drafts on reload.

Device panel design (docs/design/devices.md), ported from T3 Code's Device
panel without its cloud, pool and SSH hosts. Contracts: device schemas, the
device channel over WS RPC, a WsDeviceRpcGroup, enableDevices and
agentDeviceAccess settings, and the deviceTools session flag.
… device channel

The Device tab reaches the device hub only through the app's WS RPC
connection. hubTransport turns the device channel RPCs into a fetch that
answers a streaming Response and a WebSocket-shaped socket, reassembles
message parts, credits a chunk once its reader took it and never reopens a
channel on its own.

The stream client is T3 Code's, ported without 3D, Duo and WebRTC, and
running on that transport: AVCC with WebCodecs, MJPEG painted to the canvas
without it on iOS, and polled screenshots with live touch on Android
without it. Also the screenshot capture and the pure state helpers
(install line, platform status, new agent sessions).
Port T3 Code's device host to one local host per server. The pinned
expo-device-hub and agent-device install on Threadlines' managed Node.js
with install scripts on. The hub runs on loopback with --require-token; its
token is read from the startup line, kept in memory and sent as a bearer on
every request. hub.json lets the next server stop a leftover hub, checked by
pid and command line.

DeviceService keeps sessions in claims.json before reporting them open,
reloads them on start, and a reactor ends them when a thread is archived,
deleted or moved. Use devices off stops the hub and daemon; agent access
off stops the daemon. The device channel carries the Device tab's requests
and sockets over WS RPC with an allowlist (no serve-sim or serve-emu /api),
256 KiB chunks, a 2 MiB credit window and a slow cap. The threadlines_device
MCP endpoint serves device_list, device_open (with a bound agent-device
launcher), device_screenshot and device_close.
Sessions started with deviceTools (Use devices and Agents can use devices
on, thread not fenced) get the threadlines_device server in Codex, Claude,
ACP, Pi, Muse and OpenCode, plus a short instruction block everywhere but
ACP. Nothing changes when the settings are off.
A loopback port whose listener is the device hub, serve-sim, serve-emu, the
agent-device daemon or one of their children is refused, whoever started
it, and left out of the local servers list. Found by command line and
process tree; nothing is ever signalled this way.
Managed Node.js for the tools, hub state folder, how sessions end, the
fence on device tools, the agent-opened signal, settings reactions, and
what the live check found about serve-sim's ports.
The WS client and the environment API get a device group: the device
calls, the state stream (followed again after a reconnect) and the device
channel, which never resubscribes. A new optional capability, devices,
lets clients hide the Device tab and its entry points from servers that
lack it. client-runtime exports its device state helpers separately from
the stream code, so the main bundle does not carry the stream.
The right panel gets a Device tab for project threads on computers with
devices. On first use it explains what each step installs and installs
nothing until Use devices is on. Then it lists the computer's simulators
and emulators, opens one for the thread (starting it when off) and shows
its screen in a flat frame sized to the device, with touch, keys and
paste, a toolbar of hardware buttons, screenshot and power off, and a
Tools drawer that only shows values the device read back.

Only the device on screen streams: a hidden tab or another thread holds
no channel. An agent opening a device brings the tab up on it for
whoever is watching the thread.
Settings gets a Devices page with Use devices, Agents can use devices
(user-only in the app guide) and the versions of the device hub and
agent-device with Update. The app guide gets places for the page and its
rows and a device-tab topic. The command palette gets Open device.
No device.toggle keybinding command (older clients decode the command
list strictly), the devices capability, the toolbar layout, the
device-tab topic id, how credits are counted, and the parts of T3's
viewer not ported.
…ts exact bytes

The web hides the Device tab unless the server reports the devices
capability. An http channel counted base64 slices by estimate while the
client credits exact decoded bytes, so the window crept shut over hours.
expo-device-hub's device list leaves out simulators with no usage
timestamps. serve-sim's grid lists them; add the ones missing. Ported
from T3 Code's fix (pingdotgg/t3code#28915).
simctl cannot report the foreground app, so the drawer asks the device's
serve-sim helper every two seconds while it is open. The device header
drops its truncated "started by Threadlines" note; the picker shows it.
The preview tunnel refuses this server's own device tool ports on every
platform. Launcher pruning never follows a symlink. device_open never
picks a physical phone by default. A refused channel open closes its hub
socket. Switching agent access off refuses tools at once. A session the
daemon did not confirm closing stays claimed; an open that finishes after
its thread was put away records nothing. The state stream keeps only the
newest snapshots for a stalled subscriber. The iOS screenshot fallback
keeps its temp file in the state dir. Touch screens get a Keyboard button,
and iOS types text as US-layout key presses.
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
threadlines Skipped Skipped Nov 20, 2026 7:13pm UTC

Request Review

…rame size

Device tools now read the outside-work limit on every request, as the
browser tools do, and refuse while an outside connection's work limits
the session. The viewer stops buffering a video frame, still frame or
socket message past 16 MiB and reconnects, so a compromised saved
computer cannot exhaust the client's memory.
McpInvocationScope already carries outsideLimit; drop the extra type and
build the test's caller in full.
The device listing now carries an optional model: an iOS simulator's device
type from one cached simctl call, an emulator's AVD hardware profile, or a
phone's ro.product.model. Frames will follow it instead of the name.
…accessibility tree

Duo hinge and stance commands ride the admitted iOS input socket as tagged
messages and wait for their reply. With panel sinks set, video comes from the
fixed feed of the display in front and reopens on the other panel at a
handoff; a refused panel feed falls back to the ordinary stream. Fold and
accessibility reads go over the device channel.
Taps named from the accessibility tree, Home, rotate and Tools actions
become a numbered step log; screenshots fill the draft's free slots
(none for a model without images) and the stream canvas is recorded as a
video under the 10 MiB attachment limit. Ported from T3 Code's open PR
pingdotgg/t3code#38735.
Several devices open at once as chips at the top of the Device tab, with
rename and close; only the shown one streams. A toolbar button floats the
device over the chat, dragged by its title strip and resized from its
edges, kept inside the window. Not offered in the phone layout.
…f ending the session

The session's first-frame watchdog kept running after the video moved to a
panel feed, so a silent feed stopped everything, input socket included.
…ep the plain frame, inspect labels stay in inspect

A drag from the screen in front onto the dimmed display ends where it left.
An emulator without a hardware profile keeps the plain frame unless it folds.
A label tapped in inspect mode no longer sticks after leaving it, and hover
shows the element under the pointer again.
avdmanager is Java. Android Studio ships its own Java but most people
never set JAVA_HOME, so the hub could neither list stopped emulators nor
start one, and starting failed as "Failed to allocate an emulator port".
The server now finds JAVA_HOME, Android Studio's Java or an installed JDK
and hands it to the hub; with none, Android stays available for running
emulators and the setup screen and device list say what starting needs.
…owns the model cache

The host builds device operations each time the hub is ready, including for a
hub already running. Going through a factory made once per host keeps the
simulator and phone model caches for the host's whole life, and the test now
lists through that factory the way LocalDeviceHost does.
serve-sim sends each touch to the display in front, so a release sent after a
handoff landed on the other screen. The stream client now lifts a held touch
on its own display before any Duo command goes out, and drops the rest of a
gesture when the helper hands the input over on its own; the view no longer
sends an end after a handoff.
The size rule for new attachments also dropped ones saved before it, and a
write that overflowed storage made the next write leave out every saved
attachment. Saving now lives in one function the composer and other callers
share: a saved attachment keeps its saved data whatever its size, only new
ones must fit, and after a failed write what storage still holds stays saved.
The hand-off saves the draft's attachments itself instead of waiting for
the thread's composer, which is not mounted once the user moved on, and
offers the video for download when it did not save. The step log follows
the prompt without trimming it. The hand-off tests move to the browser
suite, where local storage is real.
…o display

Stop and both caps end capture before waiting on a running Tools action,
which is still logged when it answers; the video also stops itself at its
byte budget. The video copies whichever canvas drew the last frame,
including a Duo display's own, and Duo's raw touches become steps.
The server published ready a moment before the first listing, so a fresh
Device tab could show "0 devices" until a refresh. Ready now goes out with
the listing, and the picker says "Starting device support…" or "Looking
for devices…" instead of a count while there is no listing yet.
Reading files takes a while; if the draft's attachments changed meanwhile,
the save writes nothing and answers stale, since the change that made the
newer list saves it. Before, a slow save could overwrite the newer list and
leave a freshly pasted file unsaved.
The hand-off saves again on the current list when the draft changed under
it, then keeps any of its files the draft could not save, screenshots as
well as the video, for download from the Device tab. The toast says the
recording was not all saved instead of reporting it added.
@badcuban
badcuban added this pull request to the merge queue Nov 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 20, 2026
# Conflicts:
#	apps/server/src/provider/agentContextBudget.json
@badcuban
badcuban added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 22bbeda Nov 20, 2026
23 checks passed
@badcuban
badcuban deleted the feat/ios-simulator-setup branch November 20, 2026 19:24

This branch was previously deployed

1 inactive deployment
Preview — 87a043d0 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant